A Common LISP Hypermedia Server

A World−Wide Web (WWW) server was implemented in Common LISP in order to facilitate exploratory programming in the global hypermedia domain and to provide access to complex research programs, particularly artificial intelligence systems. The server was initially used to provide interfaces for document retrieval and for email servers. More advanced applications include interfaces to systems for inductive rule learning and natural−language question answering. Continuing research seeks to more fully generalize automatic form−processing techniques developed for email servers to operate seamlessly over the Web. The conclusions argue that presentation−based interfaces and more sophisticated form processing should be moved into the clients in order to reduce the load on servers and provide more advanced interaction models for users. 1. Overview Motivations: The Introduction discusses the motivations for developing a Common LISP HTTP server. Server: The design ideas embodied in the server are overviewed in the sections Server Features, Classes of HTTP URLs, Exporting URLs, and Computed URL Example. Applications: Screen snapshots show variety of applications, ranging from document retrieval and form−processing email servers to artificial intelligence applications, such as automatic rule induction and natural− language question answering. Programming Language: Several sections provide background on Common LISP, its object system, and the Common LISP Interface Manager. Related Systems: A system for distributing documents is briefly reviewed in the COMLINK section because it uses the Common LISP HTTP server as a WWW interface and because it embodies form−processing technology useful for the Web. Recommendations: The conclusions recap an argument for extending HTTP/HTML+ form−processing with design concepts tested in the Common LISP Interface Manager and the COMLINK system. 2. Introduction A World−Wide Web (WWW) server was implemented in Common LISP (see the FAQs and the ANSI specification) in order to facilitate exploratory programming in the global hypermedia domain and to provide access to complex research systems over the Web. The general motivation for developing this server was to provide a computational tool that would strengthen the link between the artificial intelligence researchers and the distributed hypermedia community. As the amount of information available over the World−Wide Web grows, it will become increasingly necessary to deploy intelligent methods to store, retrieve, analyze, and present information. At the same time, high−productivity programming tools employed by AI researchers will become increasingly relevant for testing new ideas in servers before incorporating them into standards−based clients. A Common LISP HTTP server provides a bridge that allows AI researchers to plug their systems into the WWW as it affords developers of distributed hypermedia standards a vehicle through which they can import effective and relevant technologies. In this spirit, the conclusion proposes that HTML+ borrow ideas from the CLIM Presentation System in order to modernize the user interfaces in clients, upgrade the interaction paradigm, and to achieve a number of efficiency gains for servers. More specific motivations for this work in the context of the Intelligent Information Infrastructure Project at M.I.T. were to develop an HTTP server meeting these criteria: High−Productivity Programming: Since all of our research systems are programmed in Common LISP and are incrementally extensible to meet research requirements, the server should be fully accessible and able to evolve at the same pace. Multiple Transport Media: We wanted to generalize our existing technology, and so, to develop a specification and control framework that works seamlessly over email and HTTP. Automatic Form−Processing: Since we already had an automatic form−processing substrate developed in an email server application, the server should be able to apply the same technology over the Web. Dynamic HTML+ Generation: Since our research emphasizes tailoring output to meet user requirements, we needed HTML+ operators that our research systems could manipulate directly to generate HTML dynamically. The current server is excellent for rapid−prototyping precisely because it builds on Common LISP and provides a fine−grained vocabulary of operators which are easily combined and modified according to evolving application requirements and draft protocol standards. The server itself is an example of rapid prototyping as its 2700 lines of LISP were written from scratch and debugged by one person in about two weeks. A high−profile demonstration at the end of the period was well−received. More recently, an authoring tool for email−based forms was generalized to emit HTML+ (Renaud, 1994). This graphical tool is now used to author forms that can work over email and WWW. The authoring tool is important for the generalization of email forms across the Web because form−processing in email performs automatic retries when user input is syntactically or semantically incorrect. To implement automatic retries over the WWW, it is necessary to dynamically generate retry forms to explain the problem, incorporate all correct answers from the original form, and allow the user to correct the erroneous answers. Computing form retries requires datastructures beyond those normally associated with scripting languages. [Other research ( Houh, Lindblad, & Wetherall, 1994) arrives at similar conclusions concerning the limitations of scripting languages for more advanced, dynamic WWW applications.] In general, more advanced form−based applications will need the more sophisticated datastructures offered by full−featured programming languages like Common LISP. 3. Initial Applications The Common LISP HTTP Server implementation was driven initially by the desire to provide WWW access to email servers and associated document or survey databases, which were built on the COMLINK System. Development of a native server offered WWW access not just for these specific applications, but also, for the COMLINK substrate systems in general. Thus, any application developed on top of COMLINK would automatically have the World−Wide Web as a user interface. The initial applications included: Document Search: Retrieve government documents via a the taxonomy of categories used to distribute them over email (see search index and form interfaces). Standing Search URLs: Find the latest, topically−focused documents with convenient anchors that prespecify keys for dynamic searches (see publications for example). Public Access Email: Send correspondence to elected government officials through a forms interface to an email server. Electronic Publications: Subscribe to electronic publications through a forms interface to an email server. All these applications handle invalid or incoherent user input by returning dynamically generated HTML+ that explains the problem and guides the user’s efforts to resubmit. 4. Advanced Applications Two artificial intelligence applications were fielded by the end of the development period: Rule Induction: Learn if−then rules over the Web (Renaud, 1994) using an advanced datahandling environment (Mallery, 1994) and a sophisticated rule induction module (Unseld & Mallery, 1991). See snapshots of the experiment and several rules learned. Natural−Language Question Answering: Ask English questions about the M.I.T. Artificial Intelligence Laboratory to a complex natural language system (Katz, 1990; Katz, 1994). Ask an English question to see how this works. See snapshots of asking a question and viewing the answer. Integration with both of these large LISP systems was possible within several of hours of work each.